Cohort-based presentation of user interaction data

ABSTRACT

An interactive, customizable, user interaction data analysis system is disclosed. The system may be configured to provide cohort-based analysis and/or graphical visualizations of user interaction data to a system operator. User interaction data may be obtained, for example, as users interact with one or more software applications. In various embodiments, interactive and customizable visualizations and analyses provided by the system may be based on user interaction data aggregated across groups of users (also referred to as cohorts of users), across particular time frames, and/or from particular software and/or computer-based applications. According to various embodiments, the system may enable insights into, for example, user interaction patterns, the frequency of software application features accessed, the performance of various aspects of software applications, and/or crashes of software applications, among others.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/095,761, filed on Dec. 3, 2013, and titled “COHORT-BASED PRESENTATIONOF USER INTERACTION DATA,” which application claims a priority benefitunder 35 U.S.C. §119 to U.S. Provisional Patent Application No.61/887,799, filed on Oct. 7, 2013, and titled “COHORT-BASED PRESENTATIONOF USER INTERACTION DATA,” which is hereby incorporated by referenceherein in its entirety.

TECHNICAL FIELD

The present disclosure relates to systems and techniques for user dataintegration, analysis, and visualization. More specifically, the presentdisclosure relates to cohort-based user interaction data integration,analysis, presentation, and visualization.

BACKGROUND

In the area of computer-based platforms and software application, userinteraction data may be collected, analyzed, and/or presented with thegoal of understanding and improving particular aspects of userinteractions. For example, in a software application-based context, userinteraction data may include various metrics including a number of timesthat a user performs a particular action (for example, clicks a button),a frequency with which a user utilizes particular functionality, and/orvarious other types of interactions of the user with the softwareapplication, among others.

SUMMARY

Such user data (also referred to as user interaction data) may beaggregated across many users. The user interaction data may then beanalyzed by, and presented to, for example, an operator. In general, theterm “analytics” may describe one or more processes of user interactiondata collection, analysis, presentation, and/or other analysis of suchdata that may provide insights to the user. Certain systems, methods,and devices of the present disclosure provide, among other features,interactive, customizable, user interaction data analysis. Such systemsmay be configured to provide cohort-based analysis and/or graphicalvisualizations of user interaction data to a system operator. Userinteraction data is obtained, for example, as users interact with one ormore software applications. In various embodiments, interactive andcustomizable visualizations and analyses provided by the system may bebased on user interaction data aggregated across groups of users (alsoreferred to as cohorts of users), across particular time frames, and/orfrom particular software and/or computer-based applications. Accordingto various embodiments, the system may enable insights into, forexample, user interaction patterns, the frequency of softwareapplication features accessed, the performance of various aspects ofsoftware applications, and/or crashes of software applications, amongothers.

According to an embodiment, a computer system is disclosed comprising:one or more computer readable storage devices configured to store: oneor more software modules including computer executable instructions; andat least one set of user interaction data collected from interactions ofa plurality of users with a software application; and one or morehardware computer processors in communication with the one or morecomputer readable storage devices and configured to execute the one ormore software modules in order to cause the computer system to:determine, based on preferences received from an operator of thecomputer system or predefined preferences, two or more groups of theplurality of users, at least one of the groups including multiple of theplurality of users; access, from the one or more computer readablestorage devices and by the one or more hardware computer processors, theuser interaction data of the plurality of users in the two or moregroups; and generate, based on the accessed user interaction data, aninteractive user interface configured for display on an electronicdisplay of the computer system, the interactive user interface includingat least: an indication of the two or more groups; and for each of thetwo or more groups, a visualization of a relationship between one ormore features and users in the group based on the accessed userinteraction data related to interactions of users in the group with thesoftware application.

According to an embodiment, a computer system is disclosed comprising:one or more computer readable storage devices configured to store one ormore software modules including computer executable instructions; andone or more hardware computer processors in communication with the oneor more computer readable storage devices and configured to execute theone or more software modules in order to cause the computer system to:access, from one or more data stores, user interaction data collectedfrom interactions of a plurality of users with features of one or moresoftware applications; receive, from an operator of the computer systemand via a user interface of the computer system, a selection of a groupof users from the plurality of users; generate a data visualizationbased on user interaction data drawn from the set of user interactiondata and associated with the selected group of users; and display thegenerated visualization on an electronic display of the computer system.

According to an embodiment, a computer system is disclosed comprising:one or more computer readable storage devices configured to store one ormore software modules including computer executable instructions; andone or more hardware computer processors in communication with the oneor more computer readable storage devices and configured to execute theone or more software modules in order to cause the computer system to:access, from one or more data stores, a set of user interaction datacomprising data collected from interactions of users with one or moresoftware applications; determine, based on preferences received from anoperator of the computer system or predefined preferences, two or moregroups of the users, at least one of the groups including multiple ofthe users; receive, from the operator of the computer system and via auser interface of the computer system, one or more filter criteria to beapplied to the set of user interaction data; determine a filtered set ofuser interaction data comprising user interaction data from the set ofuser interaction data that meets the one or more filter criteria;determine which of the two or more groups of users include users havingassociated user interaction data from the filtered set of userinteraction data; display, on an electronic display of the computersystem, a list of the determined groups of users having associated userinteraction data from the filtered set of user interaction data; anddisplay, on the electronic display, a data visualization based on thefiltered set of user interaction data.

The systems, methods, and devices described herein each have severalaspects, no single one of which is solely responsible for its desirableattributes. Without limiting the scope of this disclosure, severalnon-limiting features will now be discussed briefly.

BRIEF DESCRIPTION OF THE DRAWINGS

The following aspects of the disclosure will become more readilyappreciated as the same become better understood by reference to thefollowing detailed description, when taken in conjunction with theaccompanying drawings.

FIGS. 1A-1C illustrate sample user interfaces of the user interactiondata analysis system, according to embodiments of the presentdisclosure.

FIGS. 2A-2C illustrate additional sample user interfaces of the userinteraction data analysis system in which user cohorts are combined,according to embodiments of the present disclosure.

FIGS. 3A-3C illustrate additional sample user interfaces of the userinteraction data analysis system in which usage data is presented intables, according to embodiments of the present disclosure.

FIGS. 4A-4B illustrate additional sample user interfaces of the userinteraction data analysis system in which usage data is presented inline graphs, according to embodiments of the present disclosure.

FIGS. 5A-5B illustrate additional sample user interfaces of the userinteraction data analysis system in which usage data is presented in bargraphs, according to embodiments of the present disclosure.

FIGS. 6A-6E illustrate additional sample user interfaces of the userinteraction data analysis system in which performance data is presented,according to embodiments of the present disclosure.

FIG. 7 illustrates an additional sample user interface of the userinteraction data analysis system in which crash data is presented,according to embodiments of the present disclosure.

FIGS. 8A-8I illustrate additional sample user interfaces of the userinteraction data analysis system in which user interaction data isfiltered, according to embodiments of the present disclosure.

FIG. 9 shows a flowchart depicting illustrative operations of the userinteraction data analysis system, according to an embodiment of thepresent disclosure.

FIGS. 10A-10B illustrate a network environment and computer systems anddevices with which various methods and systems discussed herein may beimplemented.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

In order to facilitate an understanding of the systems and methodsdiscussed herein, certain terms may be defined in this document. Suchterms should be construed to include the provided definitions, theordinary and customary meaning of the terms, and/or any other impliedmeaning for the respective terms. Accordingly, any definitions providedherein do not limit the meaning of the defined terms, but only provideexemplary definitions.

Overview

An interactive, customizable, user interaction data analysis system(“the system,” which refers to any embodiment or combination ofembodiments of user interaction data analysis systems disclosed herein)is disclosed. The system may be configured to provide cohort-basedanalysis and/or graphical visualizations of user interaction data to asystem operator (or one or more operators). User interaction data may beobtained, for example, as users interact with one or more softwareapplications (for example, as users select user interface buttons, oractivate particular functionality and/or features of the softwareapplications, among others). In various embodiments, interactive andcustomizable visualizations and analyses provided by the system may bebased on user interaction data aggregated across groups of users (alsoreferred to as cohorts of users), across particular time frames, and/orfrom particular software and/or computer-based applications. Accordingto various embodiments, the system may enable insights into, forexample, user interaction patterns, the frequency of softwareapplication features accessed, the performance of various aspects ofsoftware applications, and/or crashes of software applications, amongothers. In various embodiments, the system allows an operator to analyzeand investigate user interaction data on a user-by-user basis or agroup-by-group (cohort-by-cohort) basis.

In an embodiment, data is collected by the system from user interactionswith one or more software applications at various computing devicesand/or mobile computing devices. The system then processes the userinteraction data and provides an interactive user interface to one ormore operators (such as system administrators, information technologypersonnel, managers/engineers/others responsible for a softwareapplication being monitored, and/or other interested persons) throughwhich the user interaction data may be displayed and inputs may bereceived. In an embodiment, the system comprises software including oneor more software modules. The software modules may be stored on one ormore media devices, and may be executable by one or more processors. Thesoftware modules may include modules for collecting user interactiondata, processing the data, displaying a user interface to the operatorof the system, and/or receiving inputs from the operator.

In various embodiments, the software applications from which userinteraction data may be gathered may include, for example, operatingsystems, data analysis applications, visualization applications, mapapplications, web browser applications, embedded applications, wordprocessing or spreadsheet applications, database applications, graphicsapplications, and/or communications applications, among others.

In various embodiments, user interaction data may include, for example,usage data, performance data, and/or crash data, among other types ofdata. Usage data may include, for example, information regarding users'usage or accesses of particular features of a software application. Forexample, in an embodiment, each time a user selects a particular button,accesses a particular type of data, and/or activates a particularfeature of the software application, data regarding that user action iscollected as usage data by the system. Performance data may include, forexample, information regarding performance of a software application inresponse to interactions of the users. For example, performance data mayindicate that the software application took some amount of time (forexample, three seconds, or some other amount of time) to perform anoperation, such as access a particular piece of data or perform acalculation, in response to an input from a user. Crash data mayinclude, for example, information regarding crashes of a softwareapplication. Crashes of the software application may include, forexample, instances in which the software application becomes unusable,improperly no longer accepts user input, and/or generally functionsimproperly or requires restarting.

In various embodiments, the interactive user interface of the systemincludes user interaction data displayed in the form of tables, graphs,and/or bar graphs, among others. A table may display, for example, thenumber of times a user and/or group of users accesses and/or interactswith particular features of the software application(s). In anotherexample, a table may indicate the performance of the softwareapplication(s) in connection with a user and/or group of users accessingand/or interacting with particular features of the softwareapplication(s). In yet another example, a graph and/or bar graph mayindicate user and/or user-group usage of particular features of thesoftware application(s) over time. In all of the above examples, theinteractions of particular users and/or groups of users may be comparedto one another. Additionally, in various embodiments, an operator of thesystem may specify particular groups of interest for comparison, mayspecify particular software application features of interest, and/or mayspecify particular time frames of interest, among other options.

In an embodiment, user interaction data may be organized by user. Forexample, user interaction data may be collected from one or moresoftware applications in which users of the software applications may beidentified by a username and/or some other unique identifier. Thecollected user interaction data may then be stored in, for example, adatabase or other data store and may be indexed according to theusernames of the users. Accordingly, user interaction data specific to aparticular user of the software applications may be accessed by thesystem, such as to provide an operator with insights into how particularusers interact with one or more software applications.

As mentioned above, user interaction data may be organized by groups ofusers. A group of users may comprise, for example, one or more users ofthe software applications mentioned above. Groups of users are generallyreferred to herein as cohorts. Cohorts may be determined automaticallyby the system and/or manually by the operator of the system. A cohortmay comprise a group of logically associated users and/or a cohort maycomprise an arbitrary group of users. Organizing users, and thereby userinteraction data, into cohorts enables, for example, analysis by thesystem and/or the operator of the system of grouped or aggregated userinteraction data. For example, the usage patterns of a first cohort mayeasily be compared to a second cohort. In an embodiment, any particularuser may be included in more than one cohort.

User Interaction Data Collection

As an illustrative non-limiting example, the user interaction dataanalysis system may be useful for analysis of user interactions with aword processing application. In this example, usage data of interest mayinclude, for example, a number times a user interacts with particularfeatures of the word processing application such as: a formattingoptions window, a spellcheck feature, a font selection dropdown, a wordcount tool, a font color button, a save button, etc. Each item of usagedata may additionally include, for example, an associated user, atimestamp, and/or other contextual information useful for lateranalysis. In this example, performance data may include, for example, anamount of time the word processing application takes to perform anaction or task in response to an interaction of the user such as: a timeto complete a word count, a time to save a document, etc. Similar to theusage data above, each item of performance data may additionallyinclude, for example, an associated user, a timestamp, and/or othercontextual information useful for later analysis. In this example crashdata may include, for example, a time and cause associated with a wordprocessing application crash, as well as an associated user, a crashdump file with details related to operation of the application at thetime of the crash, and/or other contextual information useful for lateranalysis.

In an embodiment, the user interaction data (including the usage data,the performance data, and/or the crash data) associated with the wordprocessing application is recorded and/or stored in a data store orother storage medium in a database (or other data structure) and/or logfile. In an embodiment, a software application developer may writesoftware code “hooks” into the software application to cause particularevents or user interactions to be recorded by the system. For example,the developer may include a “hook” associated with an event comprisingthe user selecting a save button. Then, when the user is interactingwith the software application and selects the save button, the event isrecorded as usage data. Similarly, the developer may include “hooks” torecord software performance data, crash data, and/or other data that maybe interesting for analysis and visualization in one or more userinterfaces.

The events, items, interactions, hooks, etc., that are tracked as userinteractions by the system may be referred to generally as “features.” Afeature may comprise, for example, a button the user selects, aninterface the user interacts with, an aspect of the application the useraccesses, and/or a type of data the user accesses, among others.

In an embodiment, the system is configured to automatically record userinteraction data associated with a software application without theaddition of particular “hooks” by a developer. In an embodiment, a setof default user interactions may be recorded with respect to aparticular software application, and a developer may include “hooks” forany additional interactions that are to be tracked.

In an embodiment, the system records the user interactions as theyhappen in a log file. In this embodiment, the log file may be convertedto another format, moved to another storage location, combined withother log files (e.g., of other software applications and/or users,and/or converted to a database format, before being accessed forpresentation to an operator in a user interface. In an embodiment, userinteractions are recorded directly to a database where they may beaccessed by the system for presentation to the operator in the userinterface.

In an embodiment, user interaction data is associated with specificusers. As mentioned above, users may be identified by a username, useridentification, and/or some other identifying information. In referenceto the word processing software example, a user of the word processingsoftware application may be required to log in to the application byproviding a username. That provided username may then be associated withthe interaction data collected. In another embodiment, user identifyinginformation may be drawn from the operating system or computing deviceon which the application is running. For example, an operating systemusername, an Internet Protocol (IP) address, a MAC address, a geographiclocation (for example, GPS derived coordinates), and/or any otheridentifying information may be associated with collected userinteraction data.

In various embodiments, user interaction data may be collected for oneuser, many users, many groups of users, users in disparate geographiclocations, users of multiple software applications, users of multipleversions of software applications, user having differing access scopes(for example, differing rights to access types of data or softwarefeatures), users of multiple organizations, and/or any other types orgroups of users and software applications. In an embodiment, userinteraction data may be partitioned, encrypted, and/or otherwiseprotected according to one or more of: the source from which the userinteraction data was collected, the type of software application fromwhich the user interaction data was collected, the authority or accessscopes of the users, among others. In various embodiments, userinteraction data may be stored in one or multiple sets, in one or morelocations, and on one or more data storage devices. In an embodiment,user interaction data may be collected anonymously, and/or may not beassociated with a particular user.

In various embodiment, and as is described below, collected userinteraction data is access by the system and presented in userinterfaces to an operator for viewing and analysis. In an embodiment,the user interface of the system may be updated in real-time as userinteraction data is collected from one or more software applications.

Sample User Interfaces

Embodiments of the disclosure will now be described with reference tothe accompanying Figures, wherein like numerals refer to like elementsthroughout. The terminology used in the description presented herein isnot intended to be interpreted in any limited or restrictive manner,simply because it is being utilized in conjunction with a detaileddescription of certain specific embodiments of the disclosure.Furthermore, embodiments of the disclosure may include several novelfeatures, no single one of which is solely responsible for its desirableattributes or which is essential to practicing the embodiments of thedisclosure herein described.

Many of the following described figures illustrate sample userinterfaces of the user interaction data analysis system. In variousembodiments, the user interfaces described below may be displayed in anysuitable computer system and/or application, for example, in a webbrowser window and/or a standalone software application, among others.Additionally, the functionality and/or user interfaces of the system asshown in any of the following figures may be implemented in one or morecomputer modules and/or processors, as is described below with referenceto FIGS. 10A-10B.

FIGS. 1A-1C illustrate sample user interfaces of the user interactiondata analysis system, according to embodiments of the presentdisclosure. Referring to FIG. 1A, the example user interface 102includes an analysis selection bar 104 (including selection options:Usage 106, Performance, and Crashes), a features sidebar 108, a cohortssidebar 110, visualization selection buttons (including table selectionbutton 112), data calculation buttons (including Total calculationbutton 114), and a table including columns of features (for example,ALPR Helper, Browser, etc.) and rows of cohorts (for example, Overall,Admins, etc.). In the example, the left-most column 116 of the tableincludes cohort names (for example, in second row, a cohort named“Overall” 120), the second column 118 of the table includes cohort sizes(for example, the “Overall” cohort is shown to include 1055 users),while the remaining columns indicate data associated with variousfeatures.

In operation, in the example of FIG. 1A, the operator has selected theUsage button 106 from the analysis selection bar 104. Selection of theUsage button 106 causes the system to access usage data of the userinteraction data and display one or more visualizations related to theaccessed usage data, as is described below. As mentioned above, usagedata may include, for example, information regarding users' usage oraccesses of particular features of one or more software applications. InFIG. 1A, the various features tracked by the system are listed in thefeatures sidebar 108. The listed features include, for example, ALPRhelper, Browser, Doc Search, Document, Easy Search Helper, Filter, andGeo Search. Each of these listed feature names is associated with aparticular aspect of the tracked software application with which usershave interacted, and for which data may be displayed.

Additionally, in the example of FIG. 1A, the operator has selected thetable selection button 112. Selection of the table selection button 112causes the system to display a table visualization of the userinteraction data, as is described below.

As further shown in FIG. 1A, the operator may select one or more of thefeatures listed in features sidebar 108 by selecting the check box tothe left of each feature name. Selecting a feature causes usage dataassociated with that feature to be displayed in the visualization, inthis example the table. For example, for each feature selected, a columnmay be added to the table that displays user interaction data associatedwith each cohort in respective rows of the table. Additionally, theoperator may search for particular features by using the search box inthe features sidebar 108, and/or may scroll down to see additionallisted features by using the scrollbar of the features sidebar 108.

Various cohorts (groups of users) available for selection by theoperator are listed in the cohorts sidebar 110. Similar to the featuressidebar 108, the cohorts listed in the cohorts sidebar 110 may each beselected or deselected by the operator, searched, and/or scrolledthrough. In various embodiments, cohorts listed in the cohorts sidebar110 may be automatically created by the system and/or may be manuallycreated by the operator. For example, in an embodiment, the systemprovides an interface through which the operator may select particularusers (by selection of their associated user identifying information)and may group them into a cohort. In an embodiment, the operator mayselect a name for each cohort created. In an embodiment, particularcohorts are automatically created, while others are manually created.For example, the system may automatically include an “Overall” cohortthat includes every available user, and an “Unassigned” cohort thatincludes any users not already assigned to another cohort (besides“Overall”). In the example of FIG. 1A, the listed cohorts includeOverall, 100, 11, 77th Street, AB, and Admins. The 100 cohort isunselected, while the other listed cohorts are selected.

As shown, the selected cohorts are listed in the visualization, in thisexample, respective rows of the table, with related user interactiondata (also referred to as “interaction metrics” or “metrics” herein)shown in each column of each row. The type of metric shown in the tablemay be determined by the operator's selection of one the datacalculation buttons. In the example of FIG. 1A, the operator hasselected the Total calculation button 114. Selection of the Totalcalculation button 114 causes the system to display metrics thatindicate a total number of interactions of all users in a given cohortwith a particular feature. For example, the table indicates that theOverall cohort (shown in row two of the table) has a cohort size of1055, has had 7 interactions with the ALPR Helper feature, has had 57interactions with the Browser features, etc. As the Overall cohortincludes all users for which user interaction data is available, themetrics associated with the Overall cohort are the largest of anyavailable cohort. In another example, the table indicates that the ABcohort (shown on row four of the table) has a cohort size of 10, has had0 interactions with the ALPR Helper feature, has had 1 interaction withthe Browser features, etc.

For each of the cells of the table in FIG. 1A, the cells may be filledwith a color and/or texture that visually indicates the size ofrespective metrics. In an embodiment, for each feature/column, theportion of each cell that is filled is proportional to the largestmetric shown on the table in the particular column, with the exceptionof the metrics associated with the Overall cohort. For example, for theBrowser feature column of the table, the largest metric, 16, which isassociated with the Admins cohort, is the largest metric in that columnand, thus, the entire associated cell is filled. The cell associatedwith the Browser metric of the AB cohort, 1, is accordinglyone-sixteenth filled, as the portion filled is proportional to thelargest metric 16. Similarly, the cells associated with the Cohort Sizecolumn are filled proportional to the metric associated with the BostonTea Party cohort, 21.

Turning to FIG. 1B, dropdown 130 allows the operator to select aparticular time period from which user interaction data is to bedisplayed. For example, the operator may select to view only userinteraction data from Today (the present day), Last 7 Days, Last 30Days, This Month (the current month), or Custom Range (in which theoperator can define a start and end date). When the operator hasselected a particular date range, the range is shown at indication 130,and user interaction data displayed in the present visualization (inFIG. 1B, a table) is limited to the selected time range.

Any of the cohorts listed in the table of FIG. 1A may be selected by theoperator to cause the system to display a list of users associated withthe selected cohort, as shown in FIG. 1C. In FIG. 1C, the operator hasselected to view the users of the “AB” cohort and their associated userinteraction data. For example, table 140 lists various users (forexample, a1, a10, a2, a3, etc.) and associated metrics. The operator mayselect the “Back to cohorts” button 142 to go back to table of FIG. 1Athat lists each cohort. In an embodiment, the cohorts sidebar mayautomatically change to a users sidebar 144, which lists each of theusers of the currently viewed cohort. The operator may optionally,select or deselect one or more of the users listed in the users sidebar144. Deselection of users in the users sidebar 144 causes informationrelated to the deselected user to not be displayed in the table 140.

In various embodiments, and throughout the present disclosure, the term“visualization” may be used to refer to any aspect of the data displayedto the operator (for example, via the user interface) so as to enablethe operator to understand, analyze, and/or visualize the userinteraction data. For example, in the context of the table of FIG. 1A,the term “visualization” may refer to a cell of the table, a number ortext in a cell of the table, a fill in a cell of the table, a particularcolumn and/or row of the table, a portion of the table, a group of cellsof the table, and/or any other aspect of the data displayed to theoperator. In other contexts and in other data displays andvisualizations (as described in reference to the figures below) the term“visualization” may likewise refer to any aspect and/or group of aspectsof any chart, table, line graph, bar graph, other type of graph, and/oranimation, among other data display possibilities.

Combining Cohorts

FIGS. 2A-2C illustrate additional sample user interfaces of the userinteraction data analysis system in which user cohorts are combined,according to embodiments of the present disclosure. In variousembodiments, the operator of the system may desire to combine two ormore cohorts together to analyze and visualize user interaction dataassociated with the combined cohort. Combining cohorts may beaccomplished as shown in FIGS. 2A-2C.

The user interface of FIG. 2A include a cohorts sidebar 202, in whichtwo cohorts are currently selected at 204: AB and Admins. The operatormay select the Combine Cohorts button 206, which results in the userinterface of FIG. 2B. As shown at 210 of FIG. 2B, the user interfaceindicates that the AB and Admin cohorts are going to be combined, andprovides the operator the option of naming the combined cohorts. Theoperator may then select the “Create” button to cause the combinedcohort to be created. FIG. 2C shows the combined cohort “AB and Admin”220 added to the cohorts sidebar of the user interface. Additionally, asthe AB and Admin cohort is selected, metrics associated with thecombined cohort are shown in the table at 222. It may be observed thatthe metrics associated with the combined cohort are a sum of the metricsof each individual cohort. Additionally, in an embodiment, combinationsof cohorts listed in the cohorts sidebar may include one or moredifferentiating factors so that they may be distinguished from regularcohorts. For example, in FIG. 2C, the “AB and Admins” cohort isitalicized so that the operator may understand that that cohort iscomprised of a combination of other cohorts. In an embodiment, theoperator may select the combined cohort 222 in the table to view, forexample, a list of users related to the combined cohort, and/or a listof cohorts that make up the combined cohort.

Table Visualization

FIGS. 3A-3C illustrate additional sample user interfaces of the userinteraction data analysis system in which usage data is presented intables, according to embodiments of the present disclosure. FIG. 3Ashows user interaction data in a table view in which a Total calculationbutton 302 is selected (similar to FIG. 1A described above). FIGS. 3Band 3C show user interaction data in table views in which Uniquecalculation button 304 and Average calculation button 306 are selected,respectively.

FIG. 3A shows total count metrics for three cohorts (AB and Admins,Admins, and AB) for two features (Histogram and Share Graph). Asdescribed above, the metrics displayed when the Total calculation button302 is selected comprise a total number of user interactions, with aparticular feature, for all users in a cohort, over the specified timeperiod. FIG. 3B shows, for the same cohorts and features, unique countmetrics when the Unique calculation button 304 is selected. When theUnique calculation button 304 is selected, each metric indicates a totalnumber of unique users who interacted with the particular feature atleast once in the specified time period. FIG. 3C shows, for the samecohorts and features, average count metrics when the Average calculationbutton 304 is selected. When the average calculation button 304 isselected, each metric indicates an average number of interactions, witha particular feature, for each user, over the specified time period. Forexample, average count metrics may be calculated by dividing the totalcount by the unique count, for each respective cohort and feature.

In various embodiments, other metric calculations may be displayed inthe table view of the user interface.

Line Graph Visualization

FIGS. 4A-4B illustrate additional sample user interfaces of the userinteraction data analysis system in which usage data is presented inline graphs, according to embodiments of the present disclosure. FIG. 4Ashows that the operator has selected the line graph selection button402, which causes the system to generate a user interface that shows oneor more line graphs such as line graph 404 and line graph 406. FIG. 4Aalso shows that the operator has selected the “Show one chart per:Cohort” button 422. Accordingly, the example user interface of FIG. 4Ashows one line graph for each selected cohort. In this example, as shownin the cohorts sidebar, the selected cohorts include Overall, AB, and ABand Admin, and one line graph is generated for each selected cohort,including line graph 404 and line graph 406. The system operator mayoptionally scroll down the user interface to view the remainder of theline graph 406 (which is only partially visible in FIG. 4A), and linegraphs corresponding to additional selected cohorts.

In FIG. 4A, each of the line graphs is comprised of user interactiondata indicating cohort usage of the various selected features, over thecurrently select time period. For example, the line graph 404 showsusage data over time associated with users of the Overall cohort andeach of the selected features, Histogram and Share Graph. Line 408 showsHistogram usage over time, while line 410 shows Share Graph usage overtime. In an embodiment, the operator may hover a cursor or other inputdevice over the line graph (or otherwise select aspects of the linegraph) to view detailed information. For example, in FIG. 4A theoperator has selected the date Aug. 13, 2013 (indicated by the popupdate and line at 412), causing the system to display, in the line graphwindow 404, that on that date there were ten total interactions with theHistogram feature, and 4 total interactions with the Share Graphfeature. As described previously with respect FIGS. 3A-3C, the operatormay select any of the Total, Unique, or Average calculation buttons,which will cause the line graphs to be updated accordingly.

In an embodiment, the operator may select a “Show one chart per:Feature” button 420, which causes the user interface to be updated asshown in FIG. 4B. As shown in FIG. 4B, one line graph is now shown foreach of the selected features: Histogram and Share Graph. On each linegraph, for example line graph 424 (showing user interaction data relatedto the Histogram feature), lines are plotted showing the usagerespective feature by the various selected cohorts. In the example ofFIG. 4B, for example, line 426 indicates the usage by users in theOverall cohort, while line 428 indicated usage by users in the AB andAdmins cohort. As with the user interface of FIG. 4A, and as describedpreviously with respect FIGS. 3A-3C, the operator may select any of theTotal, Unique, or Average calculation buttons, which will cause the linegraphs to be updated accordingly.

Bar Graph Visualization

FIGS. 5A-5B illustrate additional sample user interfaces of the userinteraction data analysis system in which usage data is presented in bargraphs, according to embodiments of the present disclosure. FIG. 5Ashows that the operator has selected the bar graph selection button 502,which causes the system to generate a user interface that shows one ormore sets of bar graphs such as bar graphs 504 and bar graphs 506. FIG.4A also shows that the operator has selected the “Show one chart per:Feature” button 508. Accordingly, the example user interface of FIG. 5Ashows one set of bar graphs for each selected feature. In this example,the selected features include Histogram and Share Graph, and one set ofbar graphs is generated for each selected feature, including bar graphs504 and bar graphs 506. The system operator may optionally scroll downthe user interface to view the remainder of the bar graphs 506 (which isonly partially visible in FIG. 5A), and bar graphs corresponding toadditional selected features.

In FIG. 5A, each set of bar graphs is comprised of user interaction dataindicating feature usage by each of the selected cohorts. For eachselected cohort and each selected feature, a bar graph is generated thatshows a distribution of usage of respective features by the users ofeach cohort, over the specified time period. For example, the Histogrambar graphs 504 includes a chart that shows a distribution of Histogramusage by users of the Overall cohort over the previous 30 days. TheHistogram-Overall bar graph includes five bins including 0-7, 8-14,15-21, 22-28, and 29-31. The system places each user of the cohort in abin that corresponds to the number of days in the 30 day time periodthat the user has used the Histogram feature. Thus, theHistogram-Overall bar graph indicates to the operator that, of the 1055users in the Overall cohort, 1054 used the Histogram feature for 0-7days out of the last 30 days, and 1 used the Histogram feature for 8-14days out of the last 30 days. Similarly, of users in the AB cohort, all20 used the Histogram feature for 0-7 days out of the last 30 days.

In various embodiments, a longer or shorter time period may be specifiedby the operator of the system. In these embodiments, the system mayautomatically select more or fewer than 5 bins, and each bin may cover alonger or shorter time period than 7 days.

In an embodiment, the operator may select a “Show one chart per: Cohort”button 510, which causes the user interface to be updated as shown inFIG. 5B. As shown in FIG. 5B, a set of bar graphs may be generated foreach of the selected cohorts: Overall (indicated by 512), AB (indicatedby 514), AB and Admins, and Admins. Each set of bar graphs includes bargraphs corresponding to each selected feature. The bar graphs of FIG. 5Bdisplay user interaction data in a similar manner to that of the bargraphs of FIG. 5A.

Other Usage Views Details

In an embodiment, the usage view may display usage information relatedto types of data, or data sources, accessed by users of the softwareapplication(s). In this embodiment, particular data types and/or datasources may be listed as features, and usage may be tracked as withother types of features described above.

The various usage views and/or visualizations of the system enable anoperator to determine, for example, for particular cohorts and/orparticular features, the usage patterns associated with the softwareapplication(s). This usage information may be used to, for example,determine particular features of the software application(s) that areespecially important or heavily used (or underused) by particularcohorts, and/or to determine cohorts or users that may need trainingrelated to particular features, among other advantages.

Performance Data Visualizations

FIGS. 6A-6E illustrate additional sample user interfaces of the userinteraction data analysis system in which performance data is presented,according to embodiments of the present disclosure. FIG. 6A-6E show, forexample, that the operator of the system has selected the Performanceoption 602 on the analysis selection bar. Selection of the Performanceoption 602 causes the system to access and display performance data fromthe user interaction data, and display various visualizations of theperformance data that are usable by the operator to better understanduser interactions with the one or more software applications. In each ofFIGS. 6A-6E, the operator has selected to view performance data relatedto, among other features, Doc Search, Document, Geo Search, HistogramLoad, and Share Graph.

In an embodiment, the system accesses the user interaction data todetermine those features to list in the features sidebar when thevarious analysis modes of the system are selected by the operator. Forexample, when the operator selects the Usage analysis option, the systemmay access the user interaction data to determine a list ofusage-related features from the usage data of the user interaction data.In another example, when the operator selects the Performance analysisoption, the system may access the user interaction data to determine alist of performance-related features from the performance data of theuser interaction data. The determined list of features may then bedisplayed in the features sidebar of the user interface of the system.In an embodiment, features listed in the features sidebar when thePerformance option is selected are the same as those listed when theUsage option is selected. In another embodiment, features listed in thefeatures sidebar when the Performance option is selected may not be thesame as those listed when the Usage option is selected.

Each of FIGS. 6A-6C, and 6E list various cohorts in a cohorts sidebarthat may be selected by the operator, similar to the lists in thecohorts sidebar as described above in reference to various figures (forexample, FIG. 1A). FIG. 6D lists various users associated with aselected cohort in a users sidebar that may be selected by the operator,similar to the list in the users sidebar as described above in referenceto FIG. 1C).

Similar to the tables of FIGS. 3A-3C described above, the performancedata in each of FIGS. 6A-6E is displayed in tables in which variousperformance data is organized according to selected features andcohorts. The rows of the table correspond to the selected cohorts (or inthe example of FIG. 6D, the rows correspond to users), while the columnsof the table correspond to the selected features. Cells of the tableindicate performance data associated with respective cohorts (or users)and features.

Each of FIGS. 6A-6E additionally include “Absolute” and “Relative” datadisplay options (indicated by 604 of FIG. 6A) that are alternativelyselectable by the operator, and “50th,” “90th,” “99th” percentileoptions (indicated by 606 of FIG. 6A) that are also alternativelyselectable by the operator. Each of the data display options andpercentile options may be selected by the operator of the system by, forexample, clicking on or otherwise selecting the desired option throughthe user interface. Currently selected options are indicated in FIGS.6A-6E by a shading or cross-hatching. In other embodiments, optionselection may be indicated by, for example, a coloring, texture, orother differentiating characteristic. In an embodiment, other percentileoption may be available to the operator, for example, 10th, 20th, 30th,40th, 60th, 70th, 80th, or any other percentile.

In an embodiment, selection of the “Absolute” data display option causesperformance data to be displayed that is aggregated across userinteractions for a particular time period. Selection of the “Relative”data display option causes performance data to be displayed as arelative comparison of user interaction data for one time period to userinteraction data for another time period. In an embodiment, selection ofthe various percentile options causes performance data to be displayedthat indicates, for example, performance values (also referred to asmeasurements or numbers) below which a given percentage of performancedata measurements in respective groups of measurements (for example, agroup of measurements corresponding to a particular cohort's use of aparticular feature, during a particular period of time) fall. Forexample, selection of “50th” percentile causes the system to displayperformance data that indicates 50th percentile, or median, performancevalues. In another example, selection of “90th” percentile causes thesystem to display performance data that indicates 90th percentileperformance values. In an embodiment, higher percentiles may generallyindicate worse performance values, as a higher percentile includes alarger range of performance values. Examples of performance valuesassociated with percentile options are be described in detail inreference to FIGS. 6A-6E below.

Referring now specifically to FIG. 6A, as indicated by 604 the operatorhas selected the “Absolute” performance data option and the “50th”percentile option. The table shows the performance data associated withthe selected features and cohorts. For example, for the Overall cohortand the Doc Search feature, 608 indicates that the performance of thefeature is 3.78 seconds (for example, the median performance of the DocSearch feature is 3.78 seconds). Additionally, 610 indicates that 6actions, or user interactions, are being taken into account indetermining the performance 50th percentile value. Similarly, for theOverall cohort, the median performance for the Document feature is 0.14seconds, and 114 interactions are taken into account. In an embodiment,interactions taken into account for any particular performance value aredetermined based on the total number of user interactions associatedwith a particular specified time period, cohort, and feature.

Referring to FIG. 6B, as indicated by 620 the operator has changed thepercentile option to “90th.” As may be observed, the performance numbersdisplayed in the table are generally higher than were shown in FIG. 6A,as a 90th percentile of user interactions expectedly includes longerdelays and/or worse performance associated with any particular featurethan a 50th percentile of user interactions.

Referring to FIG. 6C, as indicated by 630 the operator has changed thepercentile option to “99th.” As may be observed, the performance numbersdisplayed in the table are generally higher than were shown in FIG. 6B,as a 99th percentile of user interactions expectedly includes longerdelays and/or worse performance associated with any particular featurethan a 90th percentile of user interactions.

Referring to FIG. 6D, as mentioned above, FIG. 6D shows a sample userinterface in which the operator has selected a particular cohort (inthis example, the Unassigned cohort) to view performance data associatedwith the various users contained in that cohort. As indicated by 643, aselectable list of users is displayed, and any selected users aredisplayed in the accompanying table 640 along with associatedperformance data. The operator may select the “Back to cohorts” button642 to return to the table of cohorts (for example, any of FIGS. 6A-6C).

Referring to FIG. 6E, as indicated by 650, the operator has changed thedata display option to “Relative.” The operator may further specify afirst time period (or timeframe) at 652, and a second time period (ortimeframe) at 654. Performance data from the two selected time periodsis then accessed and a comparison between the two is displayed in thetable 652. For example, for the Overall cohort and the Geo Searchfeature, the shown 0.35 seconds value indicates that the 50th-percentileperformance of the Geo Search feature was 0.35 seconds slower in thesecond time period as compared to the first time period. In anotherexample, for the April 2013 Person Search Classes cohort and the GeoSearch feature, the shown −0.46 seconds value indicates that the50th-percentile performance of the Geo Search feature was 0.46 secondsfaster in the second time period as compared to the first time period.

In an embodiment, the performance differences in the Relative displayview may be indicated by a color, or other indicator. For example,improved performance may be indicated by green (or some other color)shading in the applicable cells, which degraded performance may beindicated by red (or some other color) shading in the applicable cells.In FIG. 6E, the cells showing degraded performance are indicated by onetype of shading, while the cells showing improved performance areindicated by another type of shading.

The performance view of the system enables the operator to discover, forexample, for particular cohorts and/or particular features, whereperformance of the software application(s) is poor, or needsimprovement. The operator may then use this information to improve thesoftware application(s), improve infrastructure underlying the softwareapplication(s), and/or assist or train users of the softwareapplication(s) so as to improve the performance of the softwareapplication(s), among other advantages.

Crashes Data Visualizations

FIG. 7 illustrates an additional sample user interface of the userinteraction data analysis system in which crash data is presented,according to embodiments of the present disclosure. FIG. 7 shows thatthe operator has selected the crashes selection button 702, which causesthe system to generate a user interface that shows a table 704 of crashdata. The table 704 include three columns that display, for each crashevent, an associated username (for example, an identifier associatedwith the user that was using the software application when a crashoccurred), a cause of the crash event, and a date and time associatedwith the crash. The table 704 may be sorted by any of the three columns.Further, the operator may select a time period for which crash data isto be displayed. In an embodiment, the cause column may indicate theparticular software application and/or feature associated with thecrash.

The performance view of the system enables the operator to discover, forexample, for particular user and/or particular features, when crashesoccur. The operator may then use this information to detect problemswith the software application(s), correct problems with the softwareapplication(s), and/or assist users of the software application(s) thathave experienced a crash, so as to improve user experience with thesoftware application(s), among other advantages.

User Interaction Data Filtering

FIGS. 8A-8I illustrate additional sample user interfaces of the userinteraction data analysis system in which user interaction data isfiltered, according to embodiments of the present disclosure. Beginningwith FIG. 8A, as indicated by indicator 802, the operator has selected aFilter option of the user interface. In response, the user interfacedisplays, at indicator 804, a list of filters applied and the numberusers meeting the filter criteria (in FIG. 8A, no filters have beenapplied, so all users with related user interaction data for the timeperiod selected, 438, meet the criteria); at indicator 806, a selectablelist of cohorts that include users that meet the applied filter criteria(in FIG. 8A, no filters have been applied, so all cohorts are listed;selected cohorts are indicated in bold, although in other embodimentsother visual differentiation may be implemented); at indicator 808, aparticular selected feature of interest (in FIG. 8A, the selectedfeature is Graph); and at indicator 810, a cohort-based visualization ofusage data associated with the users meeting the applied filter criteria(in FIG. 8A, the displayed visualization is a cohort-based graph showingtotal usage of the Graph feature by all users, for each of the selectedcohorts from the cohort list 806; in other embodiments other types ofvisualizations may be displayed).

The operator may apply a filer to the user interaction data, as shown inFIG. 8B, by selecting a “Filter” button 820. Following selection of thefilter button 820, the operator may select a particular feature uponwhich to filter from the dropdown 822. The dropdown 822 may include alist of all features for which user interaction data is available. Asshown in FIG. 8C, after the operator has selected a feature (in thisexample, the operator has selected the feature Logins (usage),indicating usage data related to the Logins feature is being used as abasis of the filtering) the user may select dropdown 830 to select, asshown in FIG. 8C, a comparison operator to be associated with theselected feature. As shown in FIG. 8E, the operator may then select anumber of interactions as 840 so as to complete the filter, and may thenselect the add button 842 to apply the filter to the user interactiondata.

As shown in FIG. 8F at indicator 850, the filter has been applied to theuser interaction data. In the example of FIG. 8F, only 4 users are shownto have greater than two interactions with the Login feature. The cohortlist 852 lists all cohorts having at least one associated user thatmeets the applied filter criteria. Three of those four cohorts areselected by the operator (Overall, NF, and Unassigned), and acorresponding visualization is displayed at 854. Additionally, the eachof the cohort listed in the cohorts list 852 includes informationindicating the percent of the total users in the cohort meeting thefilter criteria (for example, 0.91% of users in the Overall cohort havehas more than 2 interactions with the Logins feature).

In another example, as shown in FIG. 8G, the operator may apply a filterbased on performance data associated with a feature. In the exampleshown, the operator has selected Document performance data, and hasspecified the criteria to include only users have greater than oneinteraction with the Document feature, and experiencing performancebetween 0.25 and 1 seconds with respect to the Document feature. FIG. 8Hshows, at indicator 870, the filter of FIG. 8G applied to the userinteraction data. Additionally, the operator may select a differentfeature from the dropdown 872 to be the basis of the displayedvisualization. For example, the operator may choose to display avisualization showing usage of the Browser feature by all users meetingthe applied criteria. FIG. 8I shows cohort-based usage of the Documentfeature by users that meet the applied criteria.

Other Aspects of the Various Views

In various embodiments, one or more of the usage, performance, crashes,and/or filter views of the system may include other types of charts,tables, line graphs, bar graphs, animations, or other types ofvisualizations of user interaction data.

In various embodiments, the system may include popups, pop overs, and/ortool tips that comprise, for example, a window or display of informationassociated with selected and/or hovered over elements of the userinterface. For example, a popup may automatically display when theoperator hovers a cursor over a cohort in the cohorts list and/or in avisualization. The popup may display relevant to the cohort, forexample, a number of users in the cohort, a cohort creation time, and/orother information related to the cohort.

In an embodiment, the users, cohorts, and/or features displayed in theuser interface of the user interaction data analysis system aredetermined based on one or more automatically created or user-definedfiles. For example, an automatically created or user-defined file maycomprise a comma-separated values-type file, and a user may manuallydefine, in the file, lists of features, lists of users, and/or lists tocohorts. A file defining cohorts may include, for example, a list ofcohorts, and for each cohort, a list of users to be associated with thecohort. The files may be accessed and read by the system when the userinterface is generated, for example.

Sample Operations

FIG. 9 shows a flowchart depicting illustrative operations and/orprocesses of the user interaction data analysis system, according to anembodiment of the present disclosure. In various embodiments, fewerblocks or additional blocks may be included in the processes, or variousblocks may be performed in an order different from that shown in FIG. 9.In an embodiment, one or more blocks in FIG. 9 may be performed by, orimplemented in, one or more computer modules and/or processors, as isdescribed below with reference to FIGS. 10A-10B.

As shown in FIG. 9, in an embodiment blocks 902-904 may be performed byand/or occur at one or more computing devices with which users interact.Blocks 906-914, on the other hand, may be performed by and/or occur at acomputer server of the system. These various aspects of the userinteraction data analysis system are further described below inreference to FIGS. 10A-10B.

At block 902, user interactions with one or more software applicationsare received at one or more computing devices. For example, userinteractions with, for example, a data analysis application, a mapapplication, a word processing application, or some other type ofapplication may be tracked and/or stored. At block 904, the userinteraction data is communicated to a server of the system.

At block 906, the user interaction data is received at the server. Thedata is then processed by the server at block 908. For example, the userinteraction data may be organized by software application, by time, byuser, by feature, by data type (for example, usage, performance, and/orcrash data) and/or by any other useful index. Further, metrics may beprocessed and/or analyzed, and/or cohorts of users may be determined(automatically and/or in response to inputs by an operator). At block910, a user interface is generated that displays the processed userinteraction data, as described with reference to the figures above. Forexample, features and cohorts are listed, and visualization of selectedcohorts and features are displayed on the user interface.

At block 912, the operator may interact with the user interface of thesystem in any of the ways described above. These actions are received bythe system, and at block 914, the user interface is updated in responseto the operator's actions. For example, the operator may select Usage,Performance, Crashes, or Filters view, causing the system to displayinformation associated with those views. In another example, theoperator may select and/or deselect one or more cohorts and/or features,causing the system to update the displayed visualizations.

In various embodiments, user interaction data may be received andprocessed by the system at any time and/or continuously. In anembodiment, user interaction data may be updated even as the operator isviewing the data on the user interface. For example, in an embodiment,the operator may use the system to analyze substantially real-time userinteraction data.

As mentioned above, the user interaction data analysis system isadvantageously configured to provide analysis and visualizations of userinteraction data to a system operator (or one or more operators). Invarious embodiments, interactive visualizations and analyses provided bythe system may be based on user interaction data aggregated acrosscohorts, across particular time frames, and/or from particular softwareand/or computer-based applications. According to various embodiments,the system may enable insights into, for example, user interactionpatterns, the frequency of software application features accessed, theperformance of various aspects of software applications, crashes ofsoftware applications, among others. In various embodiments, the systemallows an operator to analyze and investigate user interactions data ona user-by-user basis or a group-by-group (or cohort-by-cohort) basis.The operator may then use the insights to improve user interactions withthe software application(s) and/or to improve the softwareapplication(s), among other advantages.

Implementation Mechanisms

FIG. 10A illustrates a network environment in which the user interactiondata analysis system may operate, according to embodiments of thepresent disclosure. The network environment 1050 may include one or morecomputing devices 1052, one or more mobile computing devices 1054, anetwork 1056, an interaction server 1058, and a content data store 1060.The constituents of the network environment 1050 may be in communicationwith each other either locally or over the network 1056.

In an embodiment, the computing device(s) 1052 and/or the mobilecomputing device(s) 1054 may be any computing devices capable ofdisplaying software applications to a user and receiving input from theuser. For example, the computing device(s) 1052 and/or the mobilecomputing device(s) 1054 may include one or more of the types ofcomputer-enabled devices mentioned above, such as smartphones, tablets,laptops, and/or other types of computing devices. The computingdevice(s) 1052 and/or the mobile computing device(s) 1054 may also becapable of communicating over the network 1056, for example, to requestmedia, content, and/or application data from, and/or to provide userinteraction data to, the interaction server 1058.

In some embodiments, the computing device(s) 1052 and/or the mobilecomputing device(s) 1054 may include non-transitory computer-readablemedium storage for storing content information, app data, and/orcollected user interaction data. For example, either of the computingdevice(s) 1052 and/or the mobile computing device(s) 1054 may includeone or more software modules that may implement aspects of thefunctionality of the user interaction data analysis system. These mayinclude, for example, software application 1062 and/or user interactionmodule 1064. The software application 1062 may be configured to presentcontent to a user and receive interactions from the user. For example,the software application 1062 may comprise a web app, smartphone app,and/or tablet app, among others. The user interaction module 1064 may beconfigured to gather user interaction data as the user interacts withthe software application, and to communicate the user interaction datato the interaction server 1058 for processing and display in the systemuser interface. Additional aspects, operations, and/or functionality ofcomputing device(s) 1052 and/or the mobile computing device(s) 1054 aredescribed in further detail in reference to FIG. 10B below.

The network 1056 may be any wired network, wireless network, orcombination thereof. In addition, the network 1056 may be a personalarea network, local area network, wide area network, cable network,satellite network, cellular telephone network, or combination thereof.Protocols and components for communicating via the Internet or any ofthe other aforementioned types of communication networks are well knownto those skilled in the art of computer communications and thus, neednot be described in more detail herein.

The interaction server 1058 is a computing device, similar to thecomputing devices described above, that may perform a variety of tasksto implement the operations of the user interaction data analysissystem. The interaction server may include one or more software modules1070 that may be configured to, for example, receive user interactiondata, process user interaction data, display the user interface(including various of the visualizations described above), receiveinputs from the operator, and/or update the user interface. The userinteraction data may be received from the computing device(s) 1052and/or the mobile computing device(s) 1054 over the network 1056.Additional aspects, operations, and/or functionality of interactionserver 1058 are described in further detail in referenced to FIG. 10Bbelow.

The interaction server 1058 may be in communication with the contentdata store 1060. The content data store 1060 may store, for example,received and/or processed user interaction data, among other data. Thecontent data store 1060 may be embodied in hard disk drives, solid statememories, and/or any other type of non-transitory, computer-readablestorage medium remotely or locally accessible to the interaction server1058. The content data store 1060 may also be distributed or partitionedacross multiple storage devices as is known in the art without departingfrom the spirit and scope of the present disclosure.

In various embodiments, the system may be accessible by the operatorthrough a web-based viewer, such as a web browser. In this embodiment,the user interface may be generated by the interaction server 1058 andtransmitted to the web browser of the operator. The operator may theninteract with the user interface through the web-browser. In anembodiment, the user interface of the user interaction data analysissystem may be accessible through a dedicated software application. In anembodiment, the user interface of the user interaction data analysissystem may be accessible through a mobile computing device, such as asmartphone and/or tablet. In this embodiment, the interaction server1058 may generate and transmit a user interface to the mobile computingdevice. Alternatively, the mobile computing device may include modulesfor generating the user interface, and the interaction server 1058 mayprovide user interaction data to the mobile computing device. In anembodiment, the interaction server 1058 comprises a mobile computingdevice.

According to various embodiments, the user interaction data analysissystem and other methods and techniques described herein are implementedby one or more special-purpose computing devices. The special-purposecomputing devices may be hard-wired to perform the techniques, or mayinclude digital electronic devices such as one or moreapplication-specific integrated circuits (ASICs) or field programmablegate arrays (FPGAs) that are persistently programmed to perform thetechniques, or may include one or more general purpose hardwareprocessors programmed to perform the techniques pursuant to programinstructions in firmware, memory, other storage, or a combination. Suchspecial-purpose computing devices may also combine custom hard-wiredlogic, ASICs, or FPGAs with custom programming to accomplish thetechniques. The special-purpose computing devices may be desktopcomputer systems, server computer systems, portable computer systems,handheld devices, networking devices or any other device or combinationof devices that incorporate hard-wired and/or program logic to implementthe techniques.

Computing device(s) are generally controlled and coordinated byoperating system software, such as iOS, Android, Chrome OS, Windows XP,Windows Vista, Windows 7, Windows 8, Windows Server, Windows CE, Unix,Linux, SunOS, Solaris, iOS, Blackberry OS, VxWorks, or other compatibleoperating systems. In other embodiments, the computing device may becontrolled by a proprietary operating system. Conventional operatingsystems control and schedule computer processes for execution, performmemory management, provide file system, networking, I/O services, andprovide a user interface functionality, such as a graphical userinterface (“GUI”), among other things.

For example, FIG. 10B is a block diagram that illustrates a computersystem 1000 upon which the various systems, devices, and/or methodsdiscussed herein may be implemented. For example, some or all aspects ofcomputing system 1000 may be included in any of computing device(s)1052, mobile computing device(s) 1054, and/or interaction server 1058.In an embodiment, each of the computing device(s) 1052, mobile computingdevice(s) 1054, and interaction server 1058 is comprised of a computingsystem similar to the computer system 1000 of FIG. 10B. Computer system1000 includes a bus 1002 or other communication mechanism forcommunicating information, and a hardware processor, or multipleprocessors, 1004 coupled with bus 1002 for processing information.Hardware processor(s) 1004 may be, for example, one or more generalpurpose microprocessors.

Computer system 1000 also includes a main memory 1006, such as a randomaccess memory (RAM), cache and/or other dynamic storage devices, coupledto bus 1002 for storing information and instructions to be executed byprocessor 1004. Main memory 1006 also may be used for storing temporaryvariables or other intermediate information during execution ofinstructions to be executed by processor 1004. Such instructions, whenstored in storage media accessible to processor 1004, render computersystem 1000 into a special-purpose machine that is customized to performthe operations specified in the instructions.

Computer system 1000 further includes a read only memory (ROM) 1008 orother static storage device coupled to bus 1002 for storing staticinformation and instructions for processor 1004. A storage device 1010,such as a magnetic disk, optical disk, or USB thumb drive (Flash drive),etc., is provided and coupled to bus 1002 for storing information andinstructions.

Computer system 1000 may be coupled via bus 1002 to a display 1012, suchas a cathode ray tube (CRT), LCD display, or touch screen display, fordisplaying information to a computer user and/or receiving input fromthe user or operator. An input device 1014, including alphanumeric andother keys, is coupled to bus 1002 for communicating information andcommand selections to processor 1004. Another type of user input deviceis cursor control 1016, such as a mouse, a trackball, or cursordirection keys for communicating direction information and commandselections to processor 1004 and for controlling cursor movement ondisplay 1012. This input device typically has two degrees of freedom intwo axes, a first axis (e.g., x) and a second axis (e.g., y), thatallows the device to specify positions in a plane. In some embodiments,the same direction information and command selections as cursor controlmay be implemented via receiving touches on a touch screen without acursor.

Computing system 1000 may include modules to a user interface and thevarious other aspects of the user interaction data analysis system.These modules may include, for example, the software application 1062,the user interaction module 1064, and/or the other software module(s)1070 described above, among others. The modules may be stored in a massstorage device as executable software codes that are executed by thecomputing device(s). This and other modules may include, by way ofexample, components, such as software components, object-orientedsoftware components, class components and task components, processes,functions, attributes, procedures, subroutines, segments of programcode, drivers, firmware, microcode, circuitry, data, databases, datastructures, tables, arrays, and variables.

In general, the word “module,” as used herein, refers to logic embodiedin hardware or firmware, or to a collection of software instructions,possibly having entry and exit points, written in a programminglanguage, such as, for example, Java, Lua, C or C++. A software modulemay be compiled and linked into an executable program, installed in adynamic link library, or may be written in an interpreted programminglanguage such as, for example, BASIC, Perl, or Python. It will beappreciated that software modules may be callable from other modules orfrom themselves, and/or may be invoked in response to detected events orinterrupts. Software modules configured for execution on computingdevices may be provided on a computer readable medium, such as a compactdisc, digital video disc, flash drive, magnetic disc, or any othertangible medium, or as a digital download (and may be originally storedin a compressed or installable format that requires installation,decompression or decryption prior to execution). Such software code maybe stored, partially or fully, on a memory device of the executingcomputing device, for execution by the computing device. Softwareinstructions may be embedded in firmware, such as an EPROM. It will befurther appreciated that hardware modules may be comprised of connectedlogic units, such as gates and flip-flops, and/or may be comprised ofprogrammable units, such as programmable gate arrays or processors. Themodules or computing device functionality described herein arepreferably implemented as software modules, but may be represented inhardware or firmware. Generally, the modules described herein refer tological modules that may be combined with other modules or divided intosub-modules despite their physical organization or storage

Computer system 1000 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the computer system causes orprograms computer system 1000 to be a special-purpose machine. Accordingto one embodiment, the techniques herein are performed by computersystem 1000 in response to processor(s) 1004 executing one or moresequences of one or more modules and/or instructions contained in mainmemory 1006. Such instructions may be read into main memory 1006 fromanother storage medium, such as storage device 1010. Execution of thesequences of instructions contained in main memory 1006 causesprocessor(s) 1004 to perform the process steps described herein. Inalternative embodiments, hard-wired circuitry may be used in place of orin combination with software instructions.

The term “non-transitory media,” and similar terms, as used hereinrefers to any media that store data and/or instructions that cause amachine to operate in a specific fashion. Such non-transitory media maycomprise non-volatile media and/or volatile media. Non-volatile mediaincludes, for example, optical or magnetic disks, such as storage device1010. Volatile media includes dynamic memory, such as main memory 1006.Common forms of non-transitory media include, for example, a floppydisk, a flexible disk, hard disk, solid state drive, magnetic tape, orany other magnetic data storage medium, a CD-ROM, any other optical datastorage medium, any physical medium with patterns of holes, a RAM, aPROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip orcartridge, and networked versions of the same.

Non-transitory media is distinct from but may be used in conjunctionwith transmission media. Transmission media participates in transferringinformation between non-transitory media. For example, transmissionmedia includes coaxial cables, copper wire and fiber optics, includingthe wires that comprise bus 1002. Transmission media can also take theform of acoustic or light waves, such as those generated duringradio-wave and infra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 1004 for execution. Forexample, the instructions may initially be carried on a magnetic disk orsolid state drive of a remote computer. The remote computer can load theinstructions and/or modules into its dynamic memory and send theinstructions over a telephone line using a modem. A modem local tocomputer system 1000 can receive the data on the telephone line and usean infra-red transmitter to convert the data to an infra-red signal. Aninfra-red detector can receive the data carried in the infra-red signaland appropriate circuitry can place the data on bus 1002. Bus 1002carries the data to main memory 1006, from which processor 1004retrieves and executes the instructions. The instructions received bymain memory 1006 may optionally be stored on storage device 1010 eitherbefore or after execution by processor 1004.

Computer system 1000 also includes a communication interface 1018coupled to bus 1002. Communication interface 1018 provides a two-waydata communication coupling to a network link 1020 that may be connectedto any other interface and/or network, for example network 1056 of FIG.10A. For example, communication interface 1018 may be an integratedservices digital network (ISDN) card, cable modem, satellite modem, or amodem to provide a data communication connection to a corresponding typeof telephone line. As another example, communication interface 1018 maybe a local area network (LAN) card to provide a data communicationconnection to a compatible LAN (or WAN component to communicate with aWAN). Wireless links may also be implemented. In any suchimplementation, communication interface 1018 sends and receiveselectrical, electromagnetic or optical signals that carry digital datastreams representing various types of information.

Network link 1020 typically provides data communication through one ormore networks to other data devices. For example, network link 1020 mayprovide a connection through one or more local or non-local networks tohost computers or other data equipment operated by an Internet ServiceProvider (ISP).

In an embodiment, the network link 1020 may provide data communicationservices through the world wide packet data communication network nowcommonly referred to as the “Internet.” Communication may beaccomplished through the user of, for example, electrical,electromagnetic, and/or optical signals that carry digital data streams.The signals through the various networks and the signals on network link1020 and through communication interface 1018, which carry the digitaldata to and from computer system 1000, are example forms of transmissionmedia.

Computer system 1000 may send messages and/or receive data, includingprogram code, through the network(s), network link 1020 andcommunication interface 1018. In the Internet example, a server or othercomputer-enabled device or system may transmit a requested code for anapplication program through one or more networks and/or communicationinterface 1018.

Each of the processes, methods, and algorithms described in thepreceding sections may be embodied in, and fully or partially automatedby, code modules executed by one or more computer systems or computerprocessors comprising computer hardware. The processes and algorithmsmay be implemented partially or wholly in application-specificcircuitry.

The various features and processes described above may be usedindependently of one another, or may be combined in various ways. Allpossible combinations and subcombinations are intended to fall withinthe scope of this disclosure. In addition, certain method or processblocks may be omitted in some implementations. The methods and processesdescribed herein are also not limited to any particular sequence, andthe blocks or states relating thereto can be performed in othersequences that are appropriate. For example, described blocks or statesmay be performed in an order other than that specifically disclosed, ormultiple blocks or states may be combined in a single block or state.The example blocks or states may be performed in serial, in parallel, orin some other manner. Blocks or states may be added to or removed fromthe disclosed example embodiments. The example systems and componentsdescribed herein may be configured differently than described. Forexample, elements may be added to, removed from, or rearranged comparedto the disclosed example embodiments.

Conditional language, such as, among others, “can,” “could,” “might,” or“may,” unless specifically stated otherwise, or otherwise understoodwithin the context as used, is generally intended to convey that certainembodiments include, while other embodiments do not include, certainfeatures, elements and/or steps. Thus, such conditional language is notgenerally intended to imply that features, elements and/or steps are inany way required for one or more embodiments or that one or moreembodiments necessarily include logic for deciding, with or without userinput or prompting, whether these features, elements and/or steps areincluded or are to be performed in any particular embodiment.

The term “comprising” as used herein should be given an inclusive ratherthan exclusive interpretation. For example, a general purpose computercomprising one or more processors should not be interpreted as excludingother computer components, and may possibly include such components asmemory, input/output devices, and/or network interfaces, among others.

Any process descriptions, elements, or blocks in the flow diagramsdescribed herein and/or depicted in the attached figures should beunderstood as potentially representing modules, segments, or portions ofcode which include one or more executable instructions for implementingspecific logical functions or steps in the process. Alternateimplementations are included within the scope of the embodimentsdescribed herein in which elements or functions may be deleted, executedout of order from that shown or discussed, including substantiallyconcurrently or in reverse order, depending on the functionalityinvolved, as would be understood by those skilled in the art.

It should be emphasized that many variations and modifications may bemade to the above-described embodiments, the elements of which are to beunderstood as being among other acceptable examples. All suchmodifications and variations are intended to be included herein withinthe scope of this disclosure. The foregoing description details certainembodiments of the invention. It will be appreciated, however, that nomatter how detailed the foregoing appears in text, the invention can bepracticed in many ways. As is also stated above, it should be noted thatthe use of particular terminology when describing certain features oraspects of the invention should not be taken to imply that theterminology is being re-defined herein to be restricted to including anyspecific characteristics of the features or aspects of the inventionwith which that terminology is associated. The scope of the inventionshould therefore be construed in accordance with the appended claims andany equivalents thereof.

What is claimed is:
 1. A computer system comprising: one or morecomputer readable storage devices configured to store: one or moresoftware modules including computer executable instructions; and atleast one set of user interaction data collected from interactions of aplurality of users with a software application, the at least one set ofuser interaction data including indications of interactions of theplurality of users with two or more features of the softwareapplication; and one or more hardware computer processors incommunication with the one or more computer readable storage devices andconfigured to execute the one or more software modules in order to causethe computer system to: determine, based on preferences received from anoperator of the computer system or predefined preferences, two or moregroups of the plurality of users, at least one of the groups includingmultiple of the plurality of users; access, from the one or morecomputer readable storage devices and by the one or more hardwarecomputer processors, the user interaction data of the plurality of usersin the two or more groups; receive a selection of at least two of thetwo or more groups; and generate, based on the accessed user interactiondata and the selection of the at least two of the two or more groups, aninteractive user interface configured for display on an electronicdisplay of the computer system, wherein the interactive user interfaceincludes at least: a first portion including, for each particular groupof the at least two of the two or more groups, a respective graphindicating, over a same period of time, a number of interactions ofusers of the particular group with each of at least one of the two ormore features based on the accessed user interaction data related tointeractions of users in the particular group with the softwareapplication, a second portion including a list of groups of usersincluding at least indications of the two or more groups, and in thesecond portion, controls configured to enable the operator to select oneor more of the groups of the list of groups, wherein selecting a groupof the list causes an indication associated with the group to bedisplayed.
 2. The computer system of claim 1, wherein the interactiveuser interface further includes at least: a third portion includingindications of the two or more features of the software application. 3.The computer system of claim 2, wherein the interactive user interfacefurther includes: in the third portion, a list of the two or morefeatures, wherein each of the two or more features is individuallyselectable by the operator.
 4. The computer system of claim 1, whereinthe selection of the at least two of the two or more groups is receivedfrom the operator.
 5. The computer system of claim 1, wherein theselection of the at least two of the two or more groups is based on oneor more operator preferences and/or a previous selection received fromthe operator.
 6. The computer system of claim 1, wherein the graphindicates a number of interactions of users of the particular group witheach of the at least one of the two or more features over period of timeprovided by the operator.
 7. The computer system of claim 6, wherein thegraph comprises a line graph.
 8. The computer system of claim 6, whereinthe graph comprises a bar graph.
 9. The computer system of claim 8,wherein the bar graph indicates, for the users of the particular groupand each of the at least one of the two or more features, a distributionindicating frequencies of user interactions.
 10. The computer system ofclaim 1, wherein the graph is based on user interaction data collectedduring an operator-defined period of time.
 11. The computer system ofclaim 1, wherein the user interaction data comprises at least one ofusage data, performance data, or crash data.
 12. The computer system ofclaim 11, wherein the usage data indicates user actions including atleast one of: selections of interface elements, application componentsused and/or activated, types of information and/or data accessed, orsources of information and/or data accessed.
 13. The computer system ofclaim 11, wherein the performance data indicates timings for completionof tasks by the software application.
 14. The computer system of claim1, wherein the interactive user interface further includes one or morecontrols configured to enable the operator to create groups of theplurality of users, and wherein at least one of the two or more groupsof the plurality of users is automatically determined.
 15. The computersystem of claim 1, wherein the at least one set of user interaction datais collected as users interact with the software application, whereinthe software application includes one or more hooks, wherein in responseto at least one of an action by a particular user or a crash of thesoftware application, the one or more hooks generate a record of theaction by the particular user or the crash including at least one of: anidentifier of the particular user, a time, an item selected, anapplication component used and/or activated, or an identification ofdata accessed.
 16. A computer system comprising: one or more computerreadable storage devices configured to store: one or more softwaremodules including computer executable instructions; and at least one setof user interaction data collected from interactions of a plurality ofusers with a software application, the at least one set of userinteraction data including indications of interactions of the pluralityof users with two or more features of the software application; and oneor more hardware computer processors in communication with the one ormore computer readable storage devices and configured to execute the oneor more software modules in order to cause the computer system to:determine, based on preferences received from an operator of thecomputer system or predefined preferences, two or more groups of theplurality of users, at least one of the groups including multiple of theplurality of users; access, from the one or more computer readablestorage devices and by the one or more hardware computer processors, theuser interaction data of the plurality of users in the two or moregroups; receive a selection of at least two of the two or more features;and generate, based on the accessed user interaction data and theselection of the at least two of the two or more features, aninteractive user interface configured for display on an electronicdisplay of the computer system, wherein the interactive user interfaceincludes at least: a first portion including, for each particularfeature of the at least two of the two or more features, a respectivegraph indicating, over a same period of time, a number of interactionsof users of at least one or the two or more groups with the particularfeature based on the accessed user interaction data related tointeractions of users with the particular feature of the softwareapplication, a second portion including a list of features including atleast indications of the two or more features, and in the secondportion, controls configured to enable the operator to select one ormore of the features of the list of features, wherein selecting afeature of the list causes an indication associated with the feature tobe displayed.
 17. The computer system of claim 16, wherein theinteractive user interface further includes at least: a third portionincluding indications of the two or more features of the softwareapplication.
 18. The computer system of claim 16, wherein the selectionof the at least two of the two or more features is received from theoperator.
 19. The computer system of claim 16, wherein the selection ofthe at least two of the two or more features is based on one or moreoperator preferences and/or a previous selection received from theoperator.
 20. The computer system of claim 16, wherein the graphcomprises at least one of a line graph or a bar graph.